Hi maintainers / @algora-pbc, quick visibility note for the Isaac/AimenGPT RAG bounty: this PR targets a distinct ingestion-hardening gap in the scientific RAG flow rather than duplicating the citation/reranking/context-budgeting PRs already open.

Bounty reference: https://algora.io/isaac/bounties/clq18zr98000ejs0gt0nv7gwu

Current verification from ui/:

• npx vitest run __tests__/rag-ingest.test.ts -> 4 passed
• npx prettier --check pages/api/inject-documents.ts utils/server/rag-ingest.ts __tests__/rag-ingest.test.ts -> passed
• npx tsc --noEmit --pretty false -> passed
• npm run lint -- --file pages/api/inject-documents.ts --file utils/server/rag-ingest.ts --file __tests__/rag-ingest.test.ts -> passed
• git diff --check -> passed

The main behavior change is making uploaded research PDFs with incomplete parser metadata ingest reliably instead of crashing before retrieval can happen.

Follow-up pushed in 94474be to handle one more ingestion edge case: if a PDF parses into only blank chunks, the upload route now returns a clear 400 instead of calling Chroma with empty arrays.

Verification after the update from ui/:

• npx vitest run __tests__/rag-ingest.test.ts -> 5 passed
• npx prettier --check pages/api/inject-documents.ts utils/server/rag-ingest.ts __tests__/rag-ingest.test.ts -> passed
• npx tsc --noEmit --pretty false -> passed
• npm run lint -- --file pages/api/inject-documents.ts --file utils/server/rag-ingest.ts --file __tests__/rag-ingest.test.ts -> passed
• git diff --check -> passed

Follow-up pushed in 0bc1bd2 to preserve citation metadata in scientific RAG chunks.

What changed:

• Extracts the first DOI from chunk text and stores it as primitive Chroma metadata.
• Extracts the first publication year and defaults to 0 when absent.
• Adds a stable 16-character sourceHash derived from source path and page number so retrieved chunks can be grouped back to the originating source/page without leaking document content into IDs.
• Keeps missing values Chroma-compatible with empty string / zero defaults.

Verification:

• npx vitest run tests/rag-ingest.test.ts
• npx prettier --check pages/api/inject-documents.ts utils/server/rag-ingest.ts tests/rag-ingest.test.ts
• npx tsc --noEmit --pretty false
• npm run lint -- --file pages/api/inject-documents.ts --file utils/server/rag-ingest.ts --file tests/rag-ingest.test.ts
• git diff --check

Added a short demo video artifact for Algora/reviewer convenience:

• Demo video: https://raw.githubusercontent.com/kkudumu/aimengpt/codex/rag-ingest-hardening/ui/docs/rag-ingest-demo.mp4
• The video summarizes the PDF/RAG ingestion hardening path and the verification commands from the PR body.

This is supplemental evidence for review; the code and tests remain the source of truth.

Updated the existing demo video with narrated voiceover explaining the ingestion-hardening changes, safer PDF handling, blank-chunk filtering, citation metadata preservation, and test coverage. The PR's existing demo-video link now points to the narrated MP4.